Audio synchronisation with a tunnel matrix for time series and dynamic programming

International audiencePrecise multimodal studies require precise synchronisation between audio and video signals. However, raw audio and audio from video recordings can be out of sync for several reasons. In order to re-synchronise them, a dynamic programming (DP) approach is presented here. Traditionally, DP is performed on the rectangular distance matrix comparing each value in signal A with each value in signal B. Previous work limited the search space using for example the Sakoe Chiba Band (Sakoe and Chiba, 1978). However, the overall space of the distance matrix remains identical. Here, a tunnel matrixand its according DP-algorithm are presented. The matrix contains merely the computed distance of two signals to a pre-specified bandwidth and the computational cost is equally reduced. An example implementation demonstrates the functionality on artificial data and on data from real audio andvideo recordings

Data of German speech minorities in the Archive for Spoken German: an overview

Speech islands are historically and developmentally unique and will inevitably disappear within the next decades. We urgently need to preserve their remains and exploit what is left in order to make research on language-in-contact and historical as well as current comparative language research possible. The Archive for Spoken German (AGD) at the Institute for German Language collects, fosters and archives data from completed research projects and makes them available to the wider research community. Besides large variation corpora and corpora of conversational speech, the archive already contains a range of collections of data on German speech minorities. The latter will be outlined in this chapter. Some speech island data is already made available through the personal service of the AGD, or the database of spoken German (DGD), e.g. data on Australian German, Unserdeutsch, or German in North America. Some corpora are still being prepared for publication, but still important to document for potentially interested research projects. We therefore also explain the current problems and efforts related to the curation of speech island data, from the digitization of recordings and the collection of metadata, to the integration of transcriptions, annotations and other ways of accessing and sharing data

Aix Map Task corpus:The French multimodal corpus of task-oriented dialogue

International audienceThis paper introduces the Aix Map Task corpus, a corpus of audio and video recordings of task-oriented dialogues. It was modelled afterthe original HCRC Map Task corpus. Lexical material was designed for the analysis of speech and prosody, as described in (Astésanoet al., 2007). The design of the lexical material, the protocol and some basic quantitative features of the existing corpus are presented.The corpus was collected under two communicative conditions, one audio-only condition and one face-to-face condition. The recordingstook place in a studio and a sound attenuated booth respectively, with head-set microphones (and in the face-to-face condition with twovideo cameras). The recordings have been segmented into Inter-Pausal-Units and transcribed using transcription conventions containingactual productions and canonical forms of what was said. It is made publicly available online

Light Sheet Microscopy for Single Molecule Tracking in Living Tissue

Single molecule observation in cells and tissue allows the analysis of physiological processes with molecular detail, but it still represents a major methodological challenge. Here we introduce a microscopic technique that combines light sheet optical sectioning microscopy and ultra sensitive high-speed imaging. By this approach it is possible to observe single fluorescent biomolecules in solution, living cells and even tissue with an unprecedented speed and signal-to-noise ratio deep within the sample. Thereby we could directly observe and track small and large tracer molecules in aqueous solution. Furthermore, we demonstrated the feasibility to visualize the dynamics of single tracer molecules and native messenger ribonucleoprotein particles (mRNPs) in salivary gland cell nuclei of Chironomus tentans larvae up to 200 µm within the specimen with an excellent signal quality. Thus single molecule light sheet based fluorescence microscopy allows analyzing molecular diffusion and interactions in complex biological systems

Matching across Turns in Talk-in-Interaction: The Role of Prosody and Gesture

Understanding the design of talk-in-interaction is important in many domains, including speech technology. Although phonetic, linguistic and gestural correlates have been identified for some of the social actions that conversational participants accomplish, it is only recently that researchers have begun to take account of the immediately prior interactional context as an important factor influencing the design of a speaker’s turn. The present study explores the influence of context by focussing on characteristics of short turns produced by one speaker between turns from another speaker. The hypothesis is that the speaker designs her inserted turn as a match to the prior turn when wishing to align with the previous speaker’s agenda. By contrast, non-matching would display that the speaker is non-aligning, preferring instead to initiate a new action for example. Data are taken from the AMI corpus, focussing on the spontaneous talk of first-language English participants. Using sequential analysis, such short turns are classified as either aligning or non-aligning in accordance with definitions in the Conversation Analysis literature. The degree of prosodic similarity between the inserted turn and the prior speaker’s turn is measured using novel acoustic techniques. The results show that aligning turns are significantly more similar to the immediately preceding turn, in terms of pitch contour, than non-aligning turns. In contrast to the prosodic-acoustic analysis, the results of the gestural analysis indicate that aligning and non-aligning are differentiated by the use of distinct gestures, rather than by the matching (or non-matching) of gestures across the adjacent turns. These results support the view that choice of pitch contour is managed locally, rather than by reference to an intonational lexicon. However, this is not the case for speakers’ use of gesture. The implications of these findings for a model of talk-in-interaction are considered, along with potential applications

Generating German intonation with a trainable prosodic model

A trainable prosodic model called SFC (Superposition of Functional Contours), proposed by Holm and Bailly, is here confronted to German intonation. Training material is the publicly available Siemens Synthesis Corpus that provides spoken utterances for high-quality speech synthesis. We describe the labeling framework and first evaluation results that compares the original prosody of test sentences of this corpus with their prosodic rendering by the proposed model and state-of-the-art systems available on-line on the web